Adaptivity of Stochastic Gradient Methods for Nonconvex Optimization

Adaptivity is an important yet under-studied property in modern optimization theory. The gap between the state-of-the-art theory and current practice striking that algorithms with desirable theoretical guarantees typically involve drastically different settings of hyperparameters, such as step size schemes batch sizes, regimes. Despite appealing results, divisive strategies provide little, if any, insight to practitioners select work broadly without tweaking hyperparameters. In this work, blending “geometrization” technique introduced by [L. Lei M. I. Jordan, Proceedings 20th International Conference on Artificial Intelligence Statistics, 2017, pp. 148--156] SARAH algorithm Nguyen, J. Liu, K. Scheinberg, Takáč, 34th Machine Learning, 2613--2621], we propose geometrized for nonconvex finite-sum stochastic optimization. Our proved achieve adaptivity both magnitude target accuracy Polyak--Łojasiewicz (PL) constant, present. addition, it achieves best-available convergence rate non-PL objectives simultaneously while outperforming existing PL objectives.